Resolved -
On September 4, 2025 between 15:30 UTC and 20:00 UTC the REST API endpoints git/refs, git/refs/*, and git/matching-refs/* were degraded and returned elevated errors for repositories with reference counts over 22k. On average, the request error rate to these specific endpoints was 0.5%. Overall REST API availability remained 99.9999%. This was due to the introduction of a code change that added latency to reference evaluations and overly affected repositories with many branches, tags, or other references.
We mitigated the incident by reverting the new code.
We are working to improve performance testing and to reduce our time to detection and mitigation of issues like this one in the future.
Sep 4, 20:25 UTC
Update -
The deployment has completed and we expect customers who have been impacted to see recovery. We are continuing to monitor.
Sep 4, 20:05 UTC
Update -
We are in the process of deploying the PR to revert the change that was causing timeouts to this endpoint. We will update again once that deployment is complete.
Sep 4, 19:28 UTC
Update -
We have identified a deployed change that correlates with the increase in 5XX errors to the GitRefs REST API. This is particularly affecting requests for repos with very large numbers of commits. We are working on rolling back this change which we expect will resolve the issue.
Sep 4, 18:57 UTC
Update -
API Requests is experiencing degraded performance. We are continuing to investigate.
Sep 4, 18:52 UTC
Update -
Customers are experiencing 504 responses for some API requests for regarding repo refs/tags. We are investigating.
Sep 4, 18:18 UTC
Investigating -
We are currently investigating this issue.
Sep 4, 18:16 UTC